<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Spider</title>
</head>
<body bgcolor="#ffffff">
	<h1>Spider</h1>
	<p>The spider is a tool that is used to automatically discover new
		resources (URLs) on a particular Site. It begins with a list of URLs
		to visit, called the seeds, which depends on how the Spider is
		started. The Spider then visits these URLs, it identifies all the
		hyperlinks in the page and adds them to the list of URLs to visit and
		the process continues recursively as long as new resources are found.
	</p>

	<p>The Spider can configured and started using the <a href="../../ui/dialogs/spider.html">Spider dialogue</a>.</p>

	<p>During the processing of an URL, the Spider makes a request to
		fetch the resource and then parses the response, identifying
		hyperlinks. It currently has the following behavior when processing
		types of responses:</p>

	<h4>HTML</h4>
	Processes the specific tags, identifying links to new resources:
	<ul>
		<li>Base - Proper handling</li>
		<li>A, Link, Area - 'href' attribute</li>
		<li>Frame, IFrame, Script, Img - 'src' attribute</li>
		<li>Meta - 'http-equiv' for 'location' and 'refresh'</li>
		<li>Form - proper handling of Forms with both GET and POST
			method. The fields values are generated validly, including <a
			href="http://www.w3schools.com/html5/html5_form_input_types.asp">HTML
				5.0 input types</a>.
		</li>
		<li>Comments - Valid tags found in comments are also analyzed, if
			specified in the <a href="../../ui/dialogs/options/spider.html">Options
				Spider screen</a>
		</li>
	</ul>

	<h4>Robots.txt file</h4>
	If set in the
	<a href="../../ui/dialogs/options/spider.html">Options Spider
		screen</a>, it also analyzes the 'Robots.txt' file and tries to identify
	new resources using the specified rules. It has to be mentioned that
	the Spider does not follow the rules specified in the 'Robots.txt'
	file.

	<h4>OData Atom format</h4>
	OData content using the Atom format is currently supported. All included links 
	(relative or absolute) are processed.

	<h4>Non-HTML Text Response</h4>
	Text responses are parsed scanning for the URL pattern

	<h4>Non-Text response</h4>
	Currently, the Spider does not process this type of resources.

	<h2>Other aspects</h2>
	<ul>
	<li> When checking if an URL was already visited, the behaviour regarding how
	parameters are handled can be configured on the Spider Options screen.
	</li>
	<li> When checking if an URL was already visited, there are a few common parameters
	which are ignored: jsessionid, phpsessid, aspsessionid, utm_*
	</li>
	<li> The Spider's behaviour regarding cookies depends on how the spider is started and which options are enabled. For more
	details refer to the Spider Options screen.
	</li>
	</ul>
	<p>
		The spider is configured using the <a
			href="../../ui/dialogs/options/spider.html">Spider Options screen</a>.
	</p>

	<h2>See also</h2>
	<table>
		<tr>
			<td>&nbsp;&nbsp;&nbsp;&nbsp;</td>
			<td><a href="../../ui/overview.html">UI Overview</a></td>
			<td>for an overview of the user interface</td>
		</tr>
		<tr>
			<td>&nbsp;&nbsp;&nbsp;&nbsp;</td>
			<td><a href="concepts.html">Features</a></td>
			<td>provided by ZAP</td>
		</tr>
		<tr>
			<td>&nbsp;&nbsp;&nbsp;&nbsp;</td>
			<td><a href="../../ui/dialogs/options/spider.html">Spider
					Options screen</a></td>
			<td>for an overview of the Spider Options</td>
		</tr>
	</table>

</body>
</html>
